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Signalling System 
Inventors: Ian Michael Hosking and Aled Wynne Jones 
Summary 

The benefits of captioning systems for the hard of hearing are well understood. 
However caption systems can be costly and they often obscure part of the picture. 
This problem is particularly acute in cinemas. This invention makes use of standard 
mobile phones to receive, synchronise and display caption text. 

Problem 

A significant proportion of the population with hearing difficulties benefit from captions 
on" video images such as TV broadcasts, video tapes, DVD and movies- However, 
there are a number of problems with current solutions. With open captions systems 
where the text is displayed on screen it obscures part of the image. This presents a 
particular problem with cinema where there is a reluctance for this to happen with 
general audiences. It is also not possible to personalise content eg different 
languages. There are closed caption systems where the text is displayed on a 
separate screen. The problem with such a solution is that it adds additional cost and 
complexity and such systems have had poor take up in cinemas for this reason. 

invention 

This invention uses widely available phone and hand-held computers as the caption 
display device. The invention relates to how these can be used to successfully display 
captions that are accurately synchronised to the video/picture content. 

Caption text 

The caption text can be transmitted to the hand-held device either in advance of the 
programme content happening or simultaneously with the content. 

If it is done in advance this can be achieved by well recognised methods such as 
transfer by memory card, cable, infra-red link or over a radio communications network 
such as the mobile phone network or a wireless LAN. The caption text can be stored 
on a database and made available via the Internet for download by the methods 
descirbed above.The caption text would contain timecode related information that 
would be triggered during the programme using timer in the phone and/or with some 
form of external synchronisation- 
Real-time delivery could be by similar methods but could include transmission of the 
text information in the audio of the content using a range of acoustic transmission 
techniques such as WO! <W333>A-g . Other techniques include: simple amplitude 
and frequency modulation of tones, echo modulation, critical band encoding. Masking 
of the data sound can be performed using a variety of pyscho-acoustic approaches. At 
the required data rate for real-time transmission it is likely that it may be obtrusive to 
listeners with normal hearing therefore performing the transfer in advance is 
advantageous. 
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Synchronisation 1 

Synchronisation of the text is important to ensure that it occurs at the right point in the 
programme content. It is particularly important when the captions are loaded on to the 
hand-held device in advance. The captions could be started by the user pressing a 
button and button based control could be provided to adjust the synchronisation during 
the programme. Once initiated synchronisation would be maintained by using a timer 
in the device that triggers the captions based on related timing information associated 
with the text. This has the advantage of being simple but requires obvious cues that a 
deaf person can interpret. It also suffers from potential clock drift between the source 
and the timer in the device. A better solution is to provide cues directly to the device. 
This can be done by embedding synchronisation cues in the audio of the programme 
content. These can be decoded by using a microphone and processor found in digital 
mobile phones. Embedding cues only, rather that the text requires a significantly 
lower data rate and can be inserted and convenient points where the audio will mask 
them. The techniques for this are similar to those described above for transmission of 
the caption text. 

The synchronisation codes used can either be the same code repeated whenever the 
synchronisation is required or it can be a unique code at each synchronisation point. 
The^dvantage^f4iaving^unique^ed 

enters the film or requires the captions only at certain points (for example a user who 
can only rarely requires the language translation captions) can start captioning at any 
point during the film. This is completely in the control of the user in this method. 

It is envisaged that the synchronisation codes will only be required every minute or so 
in a preferred deployment. This results in very low obtrusiveness of the 
synchronisation codes. For lower power consumption in the hand held device it is 
possible to set up the hand held device so that it knows when to expect the 
synchronisation code approximately if it has downloaded the text and synchronisation 
choreography prior to the code being transmitted. For example, when a user enters a 
cinema they start the captioning option on their hand held device. The hand held 
device listens continuously for a synchronisation code until it receives one. The hand 
held device can then start the captioning from the point which corresponds to that 
particular synchronisation code (either the beginning or if it is a unique code then the 
unique point in the film that that corresponds to). The hand held device knows 
approximately when to expect the next synchronisation code (from the synchronisation 
choreography previously downloaded) and so it can save power by switching off the 
listening mode until a time window around the expected synchronisation point. This 
time window can be defined by the manufacturer of the hand held device or the user. 
This allows the hand held device to check its synchronisation with the film every 
minute or so in a low power mode. 

To those skilled in the art it is possible to extend this text captioning to audio 
captioning. For example, the portion of the population who are visually impaired would 
benefit from additional information about what is happening on the screen. This cannot 
be delivered through a textual interface but could be delivered through the audio 
channel of the hand held device. The synchronisation and source of the audio 
captioning would be the same as that for the text captioning and can be downloaded 
prior to viewing. The user would then put earphones on during the show so that they 
can receive the extra information. This could also be used to transmit a different 
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language if the film has been dubbed so that each listener in the cinema could have 
their preferred language playing. 



Other solutions for cues using a radio network such as a Wireless LAN or infra-red 
could also be used. Cues could also be taken from the video content. 

Display 

The display could also be performed using 'head-up display* principles and equipment 
to enable the viewer to see the content and the captions simultaneously. 

Personalisation 

One of the advantages of using a hand-held device is that the caption content can be 
varied on an individual basis. For example different languages or additional 
information eg the directors comments or the names of the actors in a film. 

Applications 

Other applications include captioning for radio, TV, recorded media, opera, theatre, 
public presentations. 
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Movie Screen 



3. Audio played ba< 
through conventional 
speakers 



© 



5. The triggers are used 
to display the correct 
caption on the screen 
of the phone. 






1 . Captions with timecodes 
or similar reference 
downloaded to the phone 
for example by the mobile 
phone network- 



2. Audio triggers 
embedded in the 
sound track of 
-the-mevie 



4. Audio triggers 
decoded via the 
microphone in the 
phone using the 
phone f s processor to 
decode the data 



A schematic outlining a system for closed captioning in cinemas using a mobile phone 
for the download of caption text, decoding of acoustic synchronisation signals and 
display of the caption text. 
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EXECUTIVE SUMMARY 

There are strong legal and commercial drivers to provide captioning and audio 
description in cinemas. However current and proposed on-screen solutions are 
unlikely to give universal access to all films In all cinemas. This is because such 
solutions are seen as obtrusive to the general audience, require significant 
infrastructure and lack flexibility regarding the caption and description content. 
This situation raises the question as to whether there are any viable, low cost, off- 
screen alternatives. 

The document describes such an alternative called Intrasonics MoviMate. This 
systems works by using standard mobile phones as a caption display and 
description playback device. The captions or descriptions are downloaded to the 
phone and precisely triggered by hidden synchronisation codes in the audio of 
the film. The advantages of this system include: 

• Any film, any time potential 

-■ — Gonteat^/ariation^ l anguag es 

• Requires no fixed infrastructure in the cinema 

• Uses standard mobile phone technology 

• Caption and descriptions can be authored using a standard PC 

• Can be used at non-cinema venues 

• Interactivity possible e.g. for advertising * 

• Potential to display synchronised interactive educational content e.g. how the 
film was made 

• Potential to provide supporting descriptions to help understand a film better 
e.g. for specialist films. 

This document compares MoviMate to the other alternative options and outlines a 
phased approach to deploying MoviMate. 
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1 BACKGROUND 



This document is a submission in response to the *A Better Picture' consultation 
paper. It addresses Measure Three - Support for Cinema Exhibition and in 
particular option k c' concerning UK-wide provision for people with sensory 
disabilities and new screens. This document summarises the aims of equal 
access, discusses the various solutions for providing access and describes 
Intrasonics MoviMate as an additional solution to the current DTS based 
proposal. 



Although the goal of equal access is obvious there is a tension between aiming 
for full or universal access (any film, any time) and partial access (some films, 
some of the time). This tension is driven by the very real practical and resource 
constraints of providing full access. This submission will argue that there is a real 
option of providing universal access in a way that addresses these constraints. 



• There is the need to meet the legal requirements of the Disability 
Discrimination Act 1 995 (DDA) which requires 'reasonable adjustment' such 
as the provision of captions and audio descriptions. This has resulted in 
individuals filing lawsuits in the US with regards to the US's comparable 
legislation Section508. 

• There is strong commercial value in addressing the needs of a significant 
majority. The RNID estimate that there are nearly 9 million people In the UK 
with hearing impairment 1 The RNIB estimate that there are over 1 million 
people in the UK that could register as being blind or partially sighted. 2 



• A large proportion of people with sensory disabilities have other disabilities as 



♦ The provision of secondary, information such as captions and descriptions has 
much wider benefit. For example it can be used for: 

• language translation 

• providing additional information and commentary for cinema education 

• making specialised films more accessible to a general audience 



1 RNID estimates based on National Study of Hearing (A Davis, Hearing in Adults, Whurr 
1995) and current general population estimates. 

2 Estimates for 1996 based on RNIB prevalence rates 
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3 INTRASONICS WIOVIMATE 



3.1 Overview 

Intrasonics MoviMate is a captioning and audio description system that uses a 
mobile phone as a portable display or playback device. Precise synchronisation is 
achieved with the movie by embedding Intrasonics Sync codes in the soundtrack 
of the movie. These codes are unobtrusive to the audience but can be decoded 
by the microphone and processor in the phone in conjunction with Intrasonics 
software running on the phone. The system is currently working on a Nokia 9210 
that is already popular with the hard of hearing as it is good for text messaging 
and runs the RNID textphone application. 



3.2 How it Works 

An Intrasonics MoviMate user would undertake the following steps: 

1 Download Captions or Audio Descriptions 

The captions or audio file can be downloaded in a number of ways, the main 
ones being: 

• Via the Internet and then to the phone from a PC using the standard software 
that comes with the phone. 

• Via the mobile network. This is fine for captions and would only take a couple 
of minutes but is currently not suitable for high quality audio descriptions 
because of the size of the audio file. 

• Via the infra-red port on the phone 

• From a memory card 

2 Synchronise 

At the beginning of the film the user simply presses a start button to start the 
captions/descriptions. The software then automatically synchronises using the 
hidden sync codes in the film soundtrack. 

MoviMate can also ensure that the phone doesn't audibly ring during the 
performance. 

3 Watch or Listen 

The user simply listens to the audio via a handsfree headset or views the 
captions on the phone's screen. Caption viewing on the phone does present the 
problem of 'split viewing'. This can be moderated by a simple lens and mirror 
arrangement that makes the caption image appear at the same focal point as the 
cinema screen. Initial designs of this have already been explored and Generics 
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has display and industrial design experts who could develop a low cost easy to 
use solution. 

Caption Creation 

The caption file format for the phone is very simple as it consists of the caption 
text and timings. This means that new captions can easily be created using a tool 
such as spreadsheet. This flexibility gives the potential to create a variety of 
different caption content. 

Audio Description Creation 

Audio description work using the common 'wav* file format. This again makes 
creation and distribution of new audio descriptions easy- 
Interactivity 

Since the solution is based on a mobile phone full interactivity can be provided, 
for example interacting with adverts or questionnaires before the film starts. 
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Any film, any time potential 

Content variation possible within a single screen e.g. multiple languages 
Requires no fixed infrastructure in the cinema (although portable lens & mirror 
systems may be desirable) 
Uses standard mobile phone technology 
Caption and descriptions can be authored using a standard PC 
Can be used at non-cinema venues and using other media types e.g. video 
and DVD 

Works with any audio system 
Interactivity possible e.g. for advertising 

Can be used to display synchronised interactive educational content e.g. how 
the film was made 

Can be used to provide supporting descriptions to help understand a film 
better e.g. for specialist films. 



3.4 Applications 

The flexibility of MoviMate allows a wide variety of secondary content to be 
created. This can be used in the following ways: 

Caption variation 

Caption content can be easily varied for example 

• Full captions 

• Sub-titles 

• Summary only i.e. key words 

• Varied highlighting and emphasis of text 

Audio description variation 

In a similar manner to captions audio content can be varied. 
Multiple-languages 

Multiple-languages are easy to support even on a single screening. It is also 
possible to switch between languages during the performance and display two 
languages simultaneously where the phone has a sufficiently large screen. 

Educational content 

The content of the captions or audio can be educational. This can either be 
general education e.g. historical points or specifically cinematic education e.g. 
how a particular scene was filmed. 

Commentary 

More general commentary can be provided such as: 

• The director's comments 

• Explanation of complex plots 

• Actor details 




3.5 Deployment Options 

To ensure a successful deployment of MoviMate a 3 phase approach is 
recommend. 

Phase 1: Technical Trial 

The aim of the technical trial is to show that MoviMate works effectively in a 
cinema environment. This will build on the testing that has already been done to 
date. In particular it will allow key stakeholders to see and critique the system In 
operation. 

Cost: E40K budgetary estimate 

Output: Report with feedback from key stakeholders 

Timing: Within 6 weeks of agreement 

Phase 2: Consumer Trial & Commercial Framework Definition 

I his phase wi n conduct a Ihuiuuyli e rid^c^d-testing-ef^ew^eRsumere^Ujse- 
the system in reality. This will include the download of captions/descriptions, m- 
cinema use and re-use of the system for other films. It will also allow for the 
discussion and agreement of an acceptable commercial framework to fund a 
sustainable system. 

The aim will to elicit feedback and improve the operational aspects of the system. 

Cost: £11 OK budgetary estimate . . 

Output: Report with operational recommendations • 
Timing: Within 1 6 weeks of agreement 



Phase 3: Commercial Deployment 

The Mn-cinema infrastructure free' nature of MoviMate will allow rapid deployment 
across the UK. However it is essential that the correct commercial framework with 
all the parties involved is agreed to ensure sustainability in the long term. This 
phase will ensure that proper consultation in undertaken and that the agreed 
approach is integrated with other initiatives. 

It will also help to fund the development of a web-site that will serve as an 
information point and download for caption and descriptions and allow the 
recommendations of the previous 2 phases to be implemented. 

Cost: Full costing can be done based on the specific requirements across the 
UK. It is important to state that this would be significantly higher than phases 1 
and 2 but should be significantly less that other solutions. 
Output: Operational in cinemas around the UK 
Timing: Starting in Q4 2002 with phased roll-out 
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Commercial Model 

This proposal outlines partial funding to help accelerate an initiative to deploy 
MovtMate. Full funding will need to be met by a combination of the stakeholders 
involved. It is envisaged that the core of this will come from exhibitors and 
distributors. It is acknowledged that other Initiatives in the industry are held back 
by the complexity of the value chain. In this case we believe that MoviMate will 
produce clear benefits at a modest cost due to its low infrastructure requirements. 

Generics would be happy to submit fully costed detailed plans for the three 
phases outlined. This would require proper consultation to ensure that all the 
stakeholders' requirements are appropriately accounted for. 
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4 OPTIONS 

The current proposal for captioning and description provision is based on the 
DTS-CSS soft subtitling system. However there is a range of options and specific 
solutions available. 



4.1 Captioning 



Caption Options 




Projector 



Main 
Projector 
(Digital only) 



Portable 
Display 
Device 



Etched 
on Film 



Rear Projection 
with seat 
mounted 
reflector 



Seat Back 
Display 



Figure -j _ Shows the main options for captioning 
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Second Projector 


DTS-CSS 


Main Projector 


Reliant on the roll-out of e-cinema 


Open Captions 


Currently limited print runs (captions etched on 
films) 


Separate Adjacent Screen 


Used for live theatre and opera where there is no 
primary display screen 


Rear Projection 


Rear Window® Captioning System 


Seat Back Display 


Costly and requires wiring on a per seat basis 


Portable Display Device 


Intrasonics™ Movimate 




4.2 Audio Descriptions 



Audio Description 
Options 



As Part of 
main audio 
playback 




Central 
Playback 



Portable Device 
(Local Playback) 



Wireless 
Link 



Wired Link 



Figure 2 - Shows the main options for audio description 






Main audio playback 


Will be heard by all 


Headphone 


Allows discrete playback 


- Central Playback 


DVS Theatrical®. Playback from DTS decoder 
with broadcast via FM radio or Infra-red 
connection. 


- Local Playback 


Intrasonics™ Movimate. Audio stored locally on 
mobile phone. 
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4.3 Evaluation Criteria 

So far we have outlined 10 different options. It is important to have evaluation 
criteria to see how these options can be combined to achieve the qoal of full 
access at a reasonable cost. a 

Below is a suggested list of criteria. 




Accessibility 



Usability 



Obtrusiveness 



Infrastructure 
Requirements 



Reliability 



Flexibility 



Safety 



This refers to the degree to which a system helps 
towards the goal of "any film, any time" 



This deals primarily the two main areas of set-up and 
viewing for the end user. 



How does the system interfere with the general 
audience for example caption overlay on the film imaqe 
and audio 'spiil' from descriptions. 



What additional infrastructure is required and is the 
infrastructure required on a per screen basis. Also 
whether a solution is reliant on another proprietary 
system e.g. an audio system 



A system needs to be easy to maintain and must not 
break easily. 



This includes the support for multiple languages, 
caption text and description variation, educational 
content and the ease with which new content can be 
added. 



Any system must meet Health & Safety requirements. 




Total lifetime costs and how the costs are distributed 
between the various parties involved. 

Expected lifetime of the system and protection against 
obsolescence. 
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4.4 Comparison of the Main Caption Options 



Accessibility 



Usability 



Limited to screens with 
required equipment 



Excellent 



Limited to screens with 
required equipment 



Good 



Obtruslveness 



nfrastructure 
Requirements 



Reliability 



Flexibility 



Poor - visible by general 
audience and obscures 
part of the screen 



Very Good 



Excellent - potential for 
any screen, any time 
Good but requires 
management of mobile 
phone by user and 
results in split viewing 
that can be overcome 
with simple lens 
arrangement 



Very good. Phone part 
can be turned off during 
the performance or set to 
a-silent-mode- 



Requires DTS sound I DTS sound system, data 
system and 2 nd projector wall, reflector panels 



Safety 



Cost 



Longevity 



Single option per 
screening e.g. language 



High 3 
Similar to DTS-CSS 



Excellent 



Very Good 



TBD 



TBD 



None in cinema 
although provision of 
phones would be . 

i referable initial! 

iGood — phones will be 
managed by end-user 



Very flexible. Multiple 
languages and content 
variation is possible for a 
single screening. 
Interactivity is also 
possible 



Very Good - people 
already take their phones 
into cinemas. 



Will be superceded by 
digital projectors 



Could be adapted for e- 
cinema 



Comparatively low cost 
due to tow Infrastructure 
requirement 



Phones are upgraded by 
the end-users. Audio- 
based triggers will work 
on any audio system. 
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Comparison of the Main Audio Description Options 



Accessibility 



Usability 



Obtruslveness 



Infrastructure 
Requirements 



Limited to screens with 
required equipment 



Very Good 



Very Good 



DTS sound system, data 
wail, reflector panels 



Reliability 



Flexibility 



Cost 



Longevity 



High 3 



Multiple audio streams 
are possible if supported 
by DTS systems 
Very Good 



TBD 



Could be adapted for e~ 
cinema 



Excellent - potential for any screen 
any time 



Good but requires management of 
mobile phone by user. Handsfree kits 
are Ideal for listening 



Very good. Phone part can be turned 
off during the performance or set to a 
silent mode 

None in cinema - although provision of 
phones would be preferable Initially. 
Requires embedding of triggers in film 
audio 



Good - phones will be managed by 
end-user 



Very flexible. Multiple languages and 
content variation is possible for a single 
screening. Interactivity is also possible 
Very Good - people already take their' 
phones Into cinemas. 



Comparatively low cost due to low 
infrastructure requirement 



Phones are upgraded by the end-users. 
Audio-based triggers will work on any 
audio system. 
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5 CONCLUSION & RECOMMENDATIONS 

We believe that the goal of universal access is attainable through the use of a 
various technologies available. MoviMate complements other systems wel 
because it is an off-screen solution and is not construed by the need for in- 
cinema infrastructure. The opens up the opportunity for sensory impajred people 
to attend any f ilm at any time. MoviMate also gives the poss.bil.ty a 
wealth of other useful information to enhance the cinema go.ng experience for all. 

We therefore recommend the phased approach outlined in the document should 
be considered for funding by the Film Council as part of .ts mrt.at.ve in th.s area. 
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Caption 


Textual representation of speech and other sound effort* 


Sub-title 


Textual representation of speech only 


Open Caption 


Captions that form a permanent part of the image and 
therefore cannot be turned off. 


W/iUacu uapuOn 


Captions that can be tumed-off 


Soft subtitling 


DTS description of their projector based system 


On-screen captions 


Captions displayed on the cinema screen, which 
includes both open and closed captions 


Off-screen captions 


Captions viewed on secondary display screen (by 
inference these are closed captions) 


Audio description 


An additional soundtrack that typically describes the 
visual elements of a film 


Intrasonics 


A technology that allows data to be hidden in regular 
audio which forms the basis for MoviMate's 
synchronisation. 
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APPENDIX A ABOUT THE GENERICS GROUP 



The Generics Group is a leading integrated technology consulting, 
development and investment organisation, with an international reputation 
for successfully commercialising emerging science and technology. We 
operate from facilities, which include state-of-the-art laboratories, in 
Cambridge. England; in Zurich, Switzerland; in Stockholm, Sweden; and in 
Boston and Baltimore in the United States. 

We operate in an environment where multi-discipline teams are positively 
encouraged to maximise cross-fertilisation of ideas and offer customised 
solutions to clients in a broad range of areas including: 



• communications 

• — busines s i nno vation — 

• engineering 

• electronics 

• fuel cell and batteries 



• life sciences 

■m — loGatiorvsensors-ancLtags- 

• materials 

• medical products 



We offer a full 'turn-key* product development for our clients following a 
structured approach to delivering innovative new products and solutions. 
Wireless technology, which ranges from acoustic through to free space 
optics, is at the core of Generics technology exploitation. 
The greater part of Generics' work is for major international corporations 
with whom we have established close long-term relationships. We also 
work with small start-up companies, including many successful spin-outs 
from our own organisation and intellectual property development. 
We are grateful to the following clients and partners for permission to 
include their names in the list below. The list is a sample only as much of 
the work we do is often confidential. 



Aimpoint (Sweden) 
Amey Datel (UK) 
Aquallsa (UK) 

Asea Brown Boveri (Switzerland) 
AstraZeneca (UK) 
A.T. Kearney (Italy) 

Beiersdorf (Germany) 
Bespak (UK) 



Nationwide Building Society (UK) 

NCR (UK) 

Nike (USA) 

Nortel Networks (UK) 

Peel Hunt (UK) 
Pfizer (USA) 
Pharmacia (Sweden) 
Philips (Netherlands) 



BOC Edwards (Netherlands) 
BT (UK) 

CarnaudMetalBox (France) 
DTI (UK) 

Elopak (Norway) 

Ericsson (Sweden and Norway) 

Esselte (Sweden and Germany) 

Ethicon (UK) 

European Commission 

Financial Times (UK) 
Flytlmer Corp (USA) 
Francotyp-Postalla (Germany) 

GlaxoSmithKllne (UK) 
Grundig (Germany) 

Haagen-Dazs (UK) 
Hasbro (USA) 
Highland Distillers (UK) 

ICI (UK) 

Invensys Metering Systems (USA) 
Jarvfe (UK) 
Lear (USA) 



Plllsbury (UK) 

Procter & Gamble (USA) 

Reckltt Benckiser (UK) 
Reuters (UK) 

Sara Lee (Netherlands) 

Schott Glas (Germany) 

ServicePower (UK) 

Siemens (Germany) 

Sirnoco (UK) 

Six Continents (UK) 

Smith & Nephew (UK) 

Sonoco (USA) 

St Jude Medical (USA) 

Suber (Switzerland) 

Tannoy (UK) 
Tesco (UK) 

Texon International (UK) 
TNO (Holland) 

Unaxis (Switzerland) 

Vibration Technology (UK) 



